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Amendments to the Claims : 

This listing of claims replaces all prior versions and listings 
of claims in the application: 

Listing of Claims : 

1. (cancelled) 

2 . (previously presented) The method of claim 12 wherein 
generating the three dimensional model comprises identifying a 
structure in the range sensor information, identifying different 
sections of the structure, selecting geometric primitives for 
the different sections of the structure based at least in part 
on input from a person regarding different shapes of the 
different sections, and parametric fitting of the geometric 
primitives to the range sensor information. 

3. (previously presented) The method of claim 2, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information. 
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4. (original) The method of claim 2, wherein the at least 
one image sensor comprises multiple image sensors, and 
generating the three dimensional model further comprises 
refining the three dimensional model based on object surfaces 
mapped from images acquired by the image sensors. 

5. (previously presented) The method of claim 12, wherein 
generating the three dimensional model further comprises 
projecting and resampling points in the range sensor information 
onto a regular grid at a user-defined resolution to produce the 
height field. 

6. (original) The method of claim 5, wherein generating 
the three dimensional model further comprises processing the 
height field using hole-filling and tessellation to generate a 
triangle mesh representation of the three dimensional model . 

7. (original) The method of claim 2, wherein the range 
sensor information comprises light detection and ranging (LIDAR) 
sensor data from an active airborne laser sensor. 

8. (previously presented) The method of claim 12, wherein 
tracking orientation information comprises tracking position and 
orientation information of the at least one image sensor by 



Page 3 of 19 



Applicant : Ulrich Neumann et al . Attorney's Docket No.: 06666-156001 

Serial No.: 10/676,377 USC-3345 
Filed: September 30, 2003 

estimating a camera pose based at least in part on three 
dimensional parameters of point and line structures visible in 
the camera image, and projecting real-time video imagery 
information comprises projecting real-time video imagery- 
information from the at least one image sensor onto the three 
dimensional model based on the tracked orientation information. 

9. (original) The method of claim 8, wherein tracking 
position and orientation information of the at least one image 
sensor further comprises processing data from a tracking sensor 
system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation sensor 
input . 

10. (previously presented) The method of claim 12, wherein 
the at least one image sensor comprises multiple image sensors, 
and projecting the real-time video imagery information comprises 
projecting multiple video streams from the multiple image 
sensors onto the three dimensional model . 

11. (cancelled) 

12. (currently amended) A method comprising: 
generating a three dimensional model of a[[n]] three 
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dimensional environment from range sensor information 
representing a height field for the environment; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein projecting the real-time video imagery information 
comprises generating a depth map image from a video sensor 
viewpoint, and projective texture mapping the real-time video 
imagery information onto the three dimensional model conditioned 
upon visibility as determined from the generated depth map 
image ; and 

wherein generating the depth map image and projective 
texture mapping the real-time video imagery information are 
performed using a one-pass approach on graphics hardware that 
supports SGI OpenGL extensions. 

13. (previously presented) The method of claim 12, wherein 
visualizing the three dimensional model comprises: 
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video-projecting onto a display screen using a stereo 
video-projector; and 

coupling a rendering viewpoint to a user's head position 
using data from a tracker. 

14. (cancelled) 

15. (previously presented) The system of claim 22, wherein 
the at least one image sensor comprises multiple image sensors. 

16. (previously presented) The system of claim 15, wherein 
the real-time video imagery information comprises pre-recorded 
real-time video imagery information. 

17. (original) The system of claim 15, wherein the dynamic 
fusion imagery projection component bases the real-time video 
imagery projection on a viewpoint separate from viewpoints 
associated with the multiple image sensors. 

18. (previously presented) The system of claim 22 wherein 
the model construction component performs operations comprising: 

identifying a structure in the range sensor information; 
identifying different sections of the structure; 
selecting geometric primitives for the different sections 
of the structure based at least in part on input from a person 
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regarding different shapes of the different sections; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 

19. (original) The system of claim 18, wherein the 
operations further comprise: 

projecting and resampling points in the range sensor 
information onto a regular grid at a user-defined resolution to 
produce the height field; and 

processing the height field using hole-filling and 
tessellation to generate a triangle mesh representation of the 
three dimensional model . 

20. (previously presented) The system of claim 22, wherein 
the at least one image sensor comprises multiple image sensors, 
and the system further comprises a model refinement component 
that refines the three dimensional model based on object 
surfaces mapped from images acquired by the image sensors . 

21. (previously presented) The system of claim 22, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain position and 
orientation information of the at least one image sensor, and 
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the dynamic fusion imagery projection component projects the 
real-time video imagery information based on the position and 
orientation information. 

22. (currently amended) An augmented virtual environment 
system comprising: 

a model construction component that generates a three 
dimensional model of a[[n]] three dimensional environment from 
range sensor information representing a height field for the 
environment ; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the visualization sub-system comprises the dynamic 
fusion imagery projection component and graphics hardware that 
supports SGI OpenGL extensions, and uses a one-pass approach on 
the graphics hardware to generate a depth map image from a video 
sensor viewpoint and projective texture map the real-time video 
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imagery information to project the real-time video imagery 
information conditioned upon visibility as determined from the 
generated depth map image. 

23. (previously presented) The system of claim 22, wherein 
the visualization sub-system comprises: 

a stereo video-projector; and 
a tracker. 

24. (cancelled) 

25. (previously presented) The method of claim 29, wherein 
the surface comprises a two dimensional surface. 

26. (original) The method of claim 25, wherein placing the 
two dimensional surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

27. (cancelled) 
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28. (cancelled) 

29. (currently amended) A method comprising: 
obtaining a three dimensional model of a[[n]] three 

dimensional environment , the three dimensional model generated 
from range sensor information representing a height field for 
the three dimensional environment ; 

identifying in real time a region in motion with respect to 
a background image in real-time video imagery information from 
at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model, the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery information; 

placing a surface that corresponds to the moving region in 
the three dimensional model; 

projecting the real-time video imagery information onto the 
three dimensional model, including the surface, based on the 
position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein identifying a region in motion in real time 
comprises subtracting the background image from the real-time 
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video imagery information, identifying a foreground object in 
the subtracted real-time video imagery information, validating 
the foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent 
image frames in the real-time video imagery information. 

30. (previously presented) The method of claim 29, further 
comprising tracking the position and orientation information of 
the at least one image sensor in the environment with respect to 
the three dimensional model in real-time. 

31. (currently amended) The method of claim 30, wherein 
obtaining a three dimensional model of a[[n]] three dimensional 
environment comprises generating the three dimensional model of 
the three dimensional environment from range sensor information 
representing a height field for tho onvironmcnt . 
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32. (cancelled) 

33. (previously presented) The system of claim 37, wherein 
the surface comprises a two dimensional surface. 

34. (original) The system of claim 33, wherein the object 
detection and tracking component places the two dimensional 
surface by performing operations comprising: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

35. (cancelled) 

36. (cancelled) 

37. (currently amended) An augmented virtual environment 
system comprising: 

an object detection and tracking component that identifies 
in real time a region in motion with respect to a background 
image in real-time video imagery information from at least one 
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image sensor having associated position and orientation 
information with respect to a three dimensional model of a three 
dimensional environment, the three dimensional model generated 
from range sensor information representing a height field for 
the three dimensional environment , the background image 
comprising a single distribution background dynamically modeled 
from a time average of the real-time video imagery information, 
and places a surface that corresponds to the moving region with 
respect to the three dimensional model; 

a dynamic fusion imagery projection component that projects 
the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position 
and orientation information; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the object detection and tracking component 
identifies the moving region by performing operations comprising 
subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the 
foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 
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wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; and 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent 
image frames in the real-time video imagery information. 

38. (previously presented) The system of claim 37, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain the position and the 
orientation information associated with the at least one image 
sensor in real time in conjunction with the real-time video 
imagery . 

39. (currently amended) The system of claim 38, further 
comprising a model construction component that generates the 
three dimensional model of [[an]] the three dimensional 
environment from range acnoor information representing a height 
field for the environment . 

40. (cancelled) 
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41. (cancelled) 

42. (cancelled) 

43. (cancelled) 

44. (cancelled) 

45. (previously presented) The method of claim 3, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

46. (previously presented) The system of claim 18, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information . 
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47. (previously presented) The system of claim 46, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

48. (cancelled) 

49. (cancelled) 
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